Fix for slow the bug tokenizer adding spaces to single id decodes#32564
Merged
itazap merged 25 commits intohuggingface:mainfrom Sep 18, 2024
Merged
Fix for slow the bug tokenizer adding spaces to single id decodes#32564itazap merged 25 commits intohuggingface:mainfrom
itazap merged 25 commits intohuggingface:mainfrom
Conversation
DuyguA
changed the title
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Member
|
cc @itazap as well! |
itazap
reviewed
Aug 27, 2024
itazap
reviewed
Aug 29, 2024
itazap
reviewed
Aug 29, 2024
Collaborator
itazap
left a comment
itazap
left a comment
There was a problem hiding this comment.
Thanks for the quick update 🤗 Thanks for merging the tests! Left a few comments about the single special token case, let me know what you think!
Contributor
Author
No worries, I'll do the changes 😉 |
Contributor
Author
|
@ArthurZucker and @LysandreJik merge time please 😉 |
itazap
reviewed
Sep 10, 2024
itazap
reviewed
Sep 10, 2024
itazap
reviewed
Sep 10, 2024
itazap
reviewed
Sep 10, 2024
itazap
reviewed
Sep 10, 2024
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
itazap
reviewed
Sep 10, 2024
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
itazap
approved these changes
Sep 10, 2024
Contributor
Author
|
Gentle ping @itazap , can we do the merge? Some commits from the main was failing this branch but looks like all fixed , can we do the merge before any more breaking changes come 😁 😁 😬 |
Collaborator
|
@DuyguA Sorry for the delay! Merged !! 🚀 Thanks for working on this 🤗 |
Contributor
|
🎉Congrats @DuyguA !This issue has really been a long journey. |
Merged
BernardZach
pushed a commit
to BernardZach/transformers
that referenced
this pull request
Dec 5, 2024
…ggingface#32564) * _decode signature change and quick return * added bunch of decoding tests * signature match and return * added tests for decoding * merged decoding test * more tests for special tokens * cosmetics * fixed param * ruffed the file * refinement for single special tokens * added test for single special tokens * slight change to test name Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * minor change test name for skip tokens Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * killed already defined var Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * minor update with vars Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * killed already defined var once more Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> --------- Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Quick fix for a bug with the tokenizer, slow tokenizers add spaces in between when the input is a single id.
Fixes #29489
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@ArthurZucker